Prometheus: How can I format the summary flied

#1

Hi all,

I have a simple question:

The summary field in PagerDuty looks as following:

Summary
[FIRING:1] KubePersistentVolumeFillingUp (https-metrics 10.180.3.216:10250 kubelet /metrics mynamespace elasticsearch-bbbb-d-jkfvg elasticsearch-aaaa mon/prom-kube-prometheus-stack-prometheus prom-kube-prometheus-stack-kubelet warning)

which does consist of the Alertname (together with its Labels).

How can I remove the labels from the Summary?
So that it does contain only the Alert name.

This is my prometheus config:

- name: pagerDuty
  pagerduty_configs:
  - send_resolved: true
    http_config:
      follow_redirects: true
    routing_key: <secret>
    url: https://events.pagerduty.com/v2/enqueue
    client: '{{ template "pagerduty.default.client" . }}'
    client_url: '{{ template "pagerduty.default.clientURL" . }}'
    description: '{{ template "pagerduty.default.description" .}}'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
    severity: error

Thank You

#2

You’ll want to customize things something like this and not use the community version included:

alertmanager:
  enabled: true
  config:
    global: 
      pager_service: "service_key"
    route:
      group_by: ['...']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 12h
      receiver: 'pagerduty_service_new'
      routes:
      - receiver: 'pagerduty_service_new'
        match_re:
          receiver: 'staging'
    receivers:
    - name: 'null'
    - name: 'pagerduty_service_new'
      webhook_configs:
        - url: 'https://api.pagerduty.com/'
      pagerduty_configs:
      - routing_key: {{ global.pager_service }}
        class: "{{ range .Alerts }}{{ .Labels.alertname }}{{ end }}"
        group: "{{ range .Alerts }}{{ .Labels.namespace }}{{ end }}"
        component: "{{ range .Alerts }}{{ .Labels.pod }}{{ end }}"
        details:
          container: "{{ range .Alerts }}{{ .Labels.container }}{{ end }}"
          region: "{{ range .Alerts }}{{ .Labels.region }}{{ end }}"
          severity: "{{ range .Alerts }}{{ .Labels.severity }}{{ end }}"
          ownership: "{{ range .Alerts }}{{ .Annotations.ownership }}{{ end }}"
          summary: "{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}"
          runbook_url: "{{ range .Alerts }}{{ .Annotations.runbook_url }}{{ end }}"
          description: "{{ range .Alerts }}{{ .Annotations.description }}{{ end }}"
1 Like
#3

Thank you for your answer.

What I am trying to achieve is how the alert is being presented in PagerDuty. The following screenshot hopefully explains what I mean:

47
Screenshot 2022-04-26 at 08.37.47.png1311×320 33.2 KB

The title and Summary fields contain a number of useless information. For example the alertname and cluster where it occurs are sufficient. But instead the labels are displayed too, I would like to have the labels only in the alert description.

#4

It seems I was able to solve my problem, with a modified description line

receivers:
- name: pagerDuty
  pagerduty_configs:
  - send_resolved: true
    http_config:
      follow_redirects: true
    routing_key: <secret>
    url: https://events.pagerduty.com/v2/enqueue
    client: '{{ template "pagerduty.default.client" . }}'
    client_url: '{{ template "pagerduty.default.clientURL" . }}'
    description: '[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonAnnotations.summary }}'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
    severity: error